(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



01) 



EP 0 961 490 A2 



(12) 



EUROPEAN PATENT APPLICATION 



(43) 


Date of publication: 


/t- * \ ■ * e l_IA>1KI CIf\f\ 

(51) Intel. 6 : H04N 5/00 




01.12.1999 Bulletin 1999/48 




/AppilCd I IUI 1 f lUIHUm. 






Dale oi niing. n.uo.iyyy 




(84) 


Designated Contracting States: 


• Lean, Andy Geng-Chyun 




AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


c/o IBM United Kingdom Ltd 




MC NL PT SE 


Winchester Hampshire S021 2JN (GB) 




Designated Extension States: 


• Schaffa, Frank Andre 




AL LT LV MK RO SI 


Hartsdale, NY 10530 (US) 






• Seidman, David Israel 


(30) 


Priority: 28.05.1998 US 86157 


New York, NY 10023 (US) 


(71) 


Applicant: International Business Machines 


(74) Representative: Boyce, Conor 




Corporation 


IBM United Kingdom Limited, 




Armonk, NY 10504 (US) 


Intellectual Property Law, 






Hursley Park 


(72) 


Inventors: 


Winchester, Hampshire S021 2JN (GB) 


• 


Chang, Shu-Ping 






Shrub Oak, NY 10588 (US) 





(54) Internet convolution audio/video server 



(57) This invention describes a system for video and 
audio distribution over the Internet. The Internet Convo- 
lution Audio/video Server (ICAVS) acts as a proxy Inter- 
net server for clients performing world wide web search- 
es for video and audio. It delivers video and audio infor- 
mation in a format customized according to the client's 



system and communication constraints, and performs 
web crawling, video caching, and format conversion 
(when necessary). The proxy serving function reduces 
the requirements on the client's system for accessing 
video or audio in a variety of formats, in addition to re- 
ducing server storage requirements and Internet band- 
width. 
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Description 

[0001 ] This invention relates to a system for delivering 
audio and video over a communications network. 
[0002] There is an increasingly large volume of video 
information available on the world wide web (WWW), 
while this presents an excellent opportunity for the dis- 
semination of video for many applications (such as re- 
mote learning), several problems prevent widespread 
access to this resource 

• In order to view these video segments, a client's 
system must have a large storage capacity and 
high-bandwidth access to the server, via the public 
Internet. 

• Except in the office environment, most clients (using 
personal computers at home) have limited system 
resources - particularly system memory (RAM), disk 
storage space, processor speed, and communica- 
tions bandwidth (limited by a 28.8 Kbps modem, or 
a 56 Kbps modem for the advanced user.) 

• In viewing video which is available on the Internet, 
the client has the option of downloading the video 
file to his local storage prior to viewing, or viewing 
the video as it is "streamed" over the Internet. Video 
downloading has the disadvantage of being prohib- 
itively slow. For example, a four-minute MPEG-2 
video clip with a bit rate of 4 Mbps (occupying 30 
MBytes) requires over two hours to download, using 
a 28.8 Kbps modem. An additional disadvantage of 
video downloading is the problems it introduces re- 
lating to the unauthorized distribution of copyrighted 
material. 

• The video streaming technology used today will not 
solve the problem of bandwidth limitation. MPEG1 
streams, for example, call for a bandwidth of 1.5 
Mbps or higher, and this bandwidth can't be sup- 
plied consistently over the Internet, with today's 
technology. These bandwidth limitations result in 
small video displays of poor quality. 

[0003] The present invention (the Internet Convolu- 
tion Audio/Video Server, ICAVS) is a proxy server to be 
connected to existing Internet (or intranet) networks for 
the distribution, to network clients, of video and audio 
material. This proxy server, as described below, has fea- 
tures which facilitate access to video over the Internet, 
even in the face of the difficulties described above, and 
customize the video delivery based on the capabilities 
of the client's system. 

[0004] The concept of a proxy server is known in the 
art, as is the concept of delivering content which is cus- 
tomized for the client. Proxies such as [1] have been 
designed for improving Web access by clients through 
the use of compression and filtering techniques, as well 
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as compression. US Patent 5,678,041 describes a 
proxy server which performs the function of controlling 
access to objectionable material, including video. 
[0005] Examples of customization of video and other 
5 content are described in IBM US patent applications SN 
08/854,227 and SN 08/854,225, corresponding to Jap- 
anese Application No. 10/118,397, both applications 
filed on May 9, 1997 to P. Capek in which customized 
program material is sent to the user. For example, the 
w (video) commercials seen by the viewer of a video pro- 
gram are customized based on a profile describing the 
viewer's interests. In contrast, the current invention cus- 
tomizes the format and bandwidth of the video which is 
sent to the user, on the basis of the user's system's ca- 
'5 pabilities and the state of the network. 

[0006] Another IBM US patent application disclosing 
the teaching of storing video information is US Patent 
No. 5,758,085 filed on August 23, 1 994, entitled, "A 
Semiconductor Base Server Providing Multimedia Infor- 
mation on Demand Over wide Area Networks". The lat- 
ter application teaches the storing of processed video 
packets in a special memory means for subsequent 
transmission through requesting clients. 
[0007] US Patent 5,673,205 describes a video data 
server which monitors the client's capabilities, and 
sends still frames in the place of video if and when this 
is necessary. US Patent 5,668,948 describes a video 
server which delivers p re-stored content {i.e., no Web 
crawling is involved) and converts only from digital for- 
mat to analog (i.e., NTSC or PAL formats.) US Patent 
5,657,461 describes a user interface to a server (not a 
proxy) which does not involve format conversion. 
[0008] There is therefore a need for an audio and vid- 
eo distribution system which provides automatic on-line 
video format and bit rate customization in response to 
client and network capabilities, for storing audio and vid- 
eo data in multiple formats, for determining client and 
network capabilities during a session with the client and 
finally for transmitting decoding capabilities to a request- 
ing client during a communications session. 
[0009] Reference [2] describes a video data server 
which monitors the client's capabilities, and sends still 
frames in the place of video if and when this is neces- 
sary. US Patent 5,668,948 describes a video server 
which delivers pre-stored content (i.e. no Web crawling 
is involved) and converts only from digital format to an- 
alog (i.e. NTSC or PAL formats.) US Patent 5,657,461 
describes a user interlace to a server (not a proxy) which 
does not involve format conversion. 
[0010] The Internet Convolution Audio/Video Server 
(ICAVS) enables clients (using any Web browser) to ac- 
cess video or audio material from a variety of servers in 
a wide variety of video encoding formats and bit rates. 
The video or audio format and bit rate which is appro- 
priate for the individual client is determined and sup- 
plied. The ICAVS can customize a video or audio's bit 
rate for a specific user or group of users (if necessary) 
with a fine granularity (e.g. an AVI format video at a bit 
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rate of 14.2 kbps.) 

[0011] The ICAVS uses Web crawler technology to 
collect Web information (Web pages and their associat- 
ed data, audio, and video files) and caches it locally for 
fast response - rather than accessing content from var- 
ious Web sites, the user gains access to the content of 
many Web sites by accessing ICAVS. Video resources 
are stored in ICAVS in two formats: in the original format 
(as it was downloaded from the original server(s)) and 
a "most common format* (MCF) copy. Note that there 
may be many original copies of the same video in the 
ICAVS, since a video may be available with different 
URL's and in different formats. The MCF copy will fre- 
quently be a low-bit rate copy, for reduced-bandwidth 
distribution. The low-bit rate copy is generated by the 
ICAVS system, using a video format conversion facility. 
For example, if a 28.8 Kbps modem connection is most 
commonly used to access the ICAVS, a video file may 
be stored in its original form and as a 20 Kbps low-bit 
rate video copy. 

[001 2] When a client session is being established, the 
server determines the client system's characteristics by 
downloading to the client a software module which re- 
turns its capabilities. If necessary, a set of software can 
be downloaded to the user's system which includes the 
necessary drivers and other programs for viewing the 
video in a given format. 

[0013] The characteristics of the server-to-client net- 
work are also ascertained during session set-up, and 
may be monitored throughout the video's transmission. 
An algorithm manages how characteristics are main- 
tained and updated. This information is then stored in a 
Client Information Database for future use. 
[0014] When the client enters a URL for web informa- 
tion access, the ICAVS determines if this URL has been 
stored locally during a previous download (i.e. a "cache 
hit"). Since the ICAVS is continually "crawling" the Web 
in search of audio/video (and other) content, and cach- 
ing it, the probability of a cache hit is high. If this is the 
case, the web content is returned immediately from the 
ICAVS. 

[0015] When the requested content is a video, since 
clients differ in their video viewing capabilities, the 
ICAVS selects the format appropriate for the client's sys- 
tem (if it has been pre-stored) or generates a copy in 
this format. This means real-time compression (re-en- 
coding and/or bit rate scaling) will be performed, if nec- 
essary, to provide not only the streaming of the video, 
but also the best possible quality of the video according 
to the bandwidth available between ICAVS and the cli- 
ent system. 

[0016] If the web page of the requested URL has not 
been stored on the ICAVS in a previous download, (i.e. 
a "cache miss"), the web page is obtained for the user 
and also stored in the ICAVS for other users' use. For 
audio or video requests, a real-time audio or video for- 
mat conversion is performed, if necessary, to generate 
an audio or video file for streaming according to user's 



available bandwidth. The audio/video file is stored in the 
ICAVS for future use in its original form and in the con- 
verted new format, i.e. the MCF copy at this time. 
[0017] In the course of time, as other users request 
5 this audio/video, the ICAVS system may determine, 
based on system statistics on user bandwidth availabil- 
ity, that the MCF copy should be replaced by a copy in 
a different format. 

[001 8] An embodiment of the invention will now be de- 
10 scribed with reference to the accompanying drawing, in 
which: 

[0019] FIG. 1 shows the architecture of the Internet 
Convolution Audio/Video Server System. This figure al- 
so shows the necessary interconnection between the 
'5 components of ICAVS. 

[0020] FIG. 2 shows the flow of control from client con- 
nection to ICAVS to the delivery of the requested con- 
tent. 

[0021] Following is the functional description of each 
architectural component of ICAVS. 

Audio/Video File Server Complex 7 

[0022] This component is the repository for all of the 
audio/video content which is stored in ICAVS. Typically, 
this would be implemented by an array of disks, include 
redundancy, and be managed by a file system. The Au- 
dio/Video File Server is managed by the Audio/Video 
Data Manager 6. 

Audio/Video Data Manager 6 

[0023] As the database manager of the Audio/Video 
File Server Complex, this component accepts audio or 
video data URL's from the ICAVS Web Server front-end 
8 and consults with the Session Manager 10 to obtain 
client bandwidth parameters. It then returns information 
describing the video to be streamed (known as "meta 
data") to the ICAVS Web server front-end to initialize the 
client's ICAVS viewer 13. The AudioA/ideo Data Man- 
ager component is a software component, implemented 
by code which executes on the ICAVS' host processor. 
[0024] The following situations, handled by the Audio/ 
Video Data Manager, may occur 

• There is a cache hit, and the video format of the 
locally-stored video satisfies the client's bandwidth 
restriction. The corresponding audio or video file is 
pulled from the AudioA/ideo File Server Complex 7 
and is passed to the Streaming Traffic Pacer 9 for 
streaming video delivery. 

• There is a cache hit, but the video format does not 
satisfy the bandwidth restriction. The audio/video 
file is then fed to the Audio/Video Format Conver- 
sion Server 3, and its output is sent to Streaming 
Traffic Pacer 9 for delivery. This output is simulta- 
neously stored in the Audio/Video File Server Com- 
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plex for future use. 

• There is a cache miss. The Audio/Video Data Man- 
ager 6 requests the Locator 2 to obtain the video 
data from the Internet and feeds its data to the 
Streaming Traffic Pacer 9 directly for delivery (as 
well as to the Audio/Video File Server Complex 7 
for caching) if the requested audio video file format 
satisfies the client's bandwidth restriction. If the 
original audio or video format does not satisfy the 
bandwidth restriction, the audio/video data is fed to 
the Audio/Video Format Conversion Server 3 direct- 
ly for conversion. Again, the output of the converted 
audio/video data is fed to both the Streaming Traffic 
Pacer and the Audio/Video File Server Complex. 
The input to the Audio/Video Format Conversion 
Server (i.e. the audio or video in its original format) 
is also stored. Depending on system design, a copy 
of the video (or audio) streaming format which is 
most commonly requested can also be produced 
and stored in the Audio/Video File Server Complex. 

Non- Audio/Video Data Manager 5 

[0025] On receiving a non-audio/video URL request 
from the ICAVS Web server 8, this component works as 
a cache manager for Web pages previously collected in 
the Web Page Database 4. The Non-AudioA/ideo Data 
Manager component is a software component, imple- 
mented by code which executes on the ICAVS* host 
processor. 

[0026] It is also the database manager of the Web 
Page Database. In the case of a cache miss, the Non- 
Audio/vldeo Data Manager requests the Locator 2 (see 
below) to obtain the requested Web page from the In- 
ternet and returns it to the server front -end. It also keeps 
a local copy in the Web Page Database 4 for future use. 

Audio/Video Format Conversion Server 3 

[0027] This is a high-performance audio/video file for- 
mat conversion server which will most likely be imple- 
mented in hardware in order to satisfy real-time require- 
ments. An array of hardware encoders and decoders 
connected with a switch will implement the conversions. 
Examples of such hardware encoders are the IBM 
MPEGSE10-30 MPEG-2 encoder chipset, the C-Cube 
Microsystems CLM4500 MPEG-1 Video Encoder and 
CLM4400 MPEG-2 Video Encoder. Software implemen- 
tation is also possible if timing restrictions are less re- 
strictive. 

Web Client 11 

[0028] This component is the client system which is 
comprised of a standard Web browser 1 2 and an ICAVS 
plug-in (Preferred ICAVS Viewer 1 3) for client system 
characterization and streaming video viewing. Standard 



web browsers include Microsoft's Windows Explorer 
and Netscape's Navigator. 

[0029] Note that the system characterization informa- 
tion which ICAVS determines includes not only commu- 

5 nication channels but also CPU type and speed, graph- 
ics capabilities, memory buffer size, etc. In order to re- 
duce the load on the ICAVS for real-time services, the 
Preferred ICAVS Viewer corresponds to the format of 
the MCF copy of the video, if the client's system can 

io support this. 

ICAVS Web Server 8 

[0030] The ICAVS Web Server is a software compo- 
15 nent, implemented by code which executes on the 
ICAVS' host processor. As a normal Web server using 
the HTTP protocol, this component should be used as 
Web proxy server to fully employ its benefit. The Web 
server on the back-end distinguishes audio/video from 
20 non-audio/video URL's and relays client requests for the 
former URL's to the AudioA/ideo Data Manager 6 and 
for the latter URL's to the Non-Audio/Video Data Man- 
ager 5 (see below.) 

[0031] The Web server also invokes a Session Man- 
25 ager 10 to contact the client in order to determine the 
bandwidth parameters between the server and client 
system and other client system characteristics. 

Session Manager 10 

30 

[0032] This component collects bandwidth informa- 
tion from the client system when the ICAVS is first con- 
tacted by the client and subsequently, on a periodic ba- 
sis, if the client is active. This component is a software 
35 component, implemented by code which executes on 
the ICAVS' host processor. 

[0033] Collected data includes bandwidth between 
ICAVS and client, client IP address, client processor 
type, memory buffer size, etc. This information is stored 
40 jn the Client Information Database 1 4 which is managed 
by the Session Manager. The Session Manager pro- 
vides bandwidth information to the Video Data Manager 
upon request, for streaming video delivery. 

45 Web Page Database 4 

[0034] This component stores non -audio/video Web 
content which has been downloaded by the Web Crawl- 
er 1 or Locator 2 (see below). Typically, this would be 
so implemented by an array of disks (including redundan- 
cy), and is managed by a database application which 
executes on the ICAVS' host processor. 

Client Information Database 14 

55 

[0035] This component stores client-specific informa- 
tion (such as bandwidth parameters, client system ca- 
pabilities, client viewer type, etc.) It is accessed by the 
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Session Manager 10 when determining the how to best 
accommodate a client's request. 
[0036] Typically, this would be implemented by an ar- 
ray of disks (including redundancy), and would be man- 
aged by a database application which executes on the 
ICAVS' host processor 

Web Crawler 1 

[0037] This component continually accesses the In- 
ternet to collect Web pages and resources for caching 
by the ICAVS. The ICAVS Web Crawler component is a 
software component, implemented by code which exe- 
cutes on the ICAVS' host processor. 
[0038] Web resources with non-video data are 
passed to the Non-Audio/Video Data Manager 5 (along 
with their URL's) for storage in the Web Page Database 
4. Video data is passed to Audio/Video Data Manager 
6, along with its URL, for further processing and storage 
in the Audio/Video File Server Complex 7. 
[0039] An audio/video quality check module may be 
implemented in the ICAVS to ensure that the video or 
audio which is cached is of acceptable quality. The 
checking performed can be similar in function to that 
which is performed on multimedia files by a player such 
as ActiveMovie prior to the start of play. 

Locator 2 

[0040] The Locator accesses the Internet for Web re- 
sources (like the Web Crawler 1 ) but does so at the re- 
quest of the Non-Audio/Video Data Manager 5, or the 
Audio/Video Data Manager 6, in real time. This normally 
happens when a client requests Web resources which 
are not cached on the ICAVS. The obtained Web data 
and its corresponding URL are returned to the request- 
ing component. 

[0041] The ICAVS Locator component is a software 
component, implemented by code which executes on 
the ICAVS' host processor. 

Streaming Traffic Pacer 9 

[0042] As with all streaming audio/video servers, this 
component paces the speed of audio/video data into the 
network. It also accepts VCR-like control signals from 
the client to manipulate streaming audio/video data. It 
can also adjust to bandwidth changes, if so desired. 
[0043] The ICAVS Streaming Traffic Pacer is a soft- 
ware component, implemented by code which executes 
on the ICAVS' host processor. 
[0044] FIG. 2 shows a flowchart of the operation of 
ICAVS. After a client connects to ICAVS 15, the state of 
the network between client and server, and, if neces- 
sary, determines the client system's capabilities and 
stores them 17 in the client information database 14. 
[0045] The client, using an Electronic Program Guide 
or other means, then requests access to a specific item 
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of audio, video, or other content 18. 
[0046] If audio or video content has been requested, 
1 9, the Non- Audio/Video Data Manager 5 look for this 
content 20 in the Web Page Database 4. If it is available 

5 there 22, the Web Server 8 Transmits it to the client 24. 
If not, the Locator 2 obtains it 23. If audio or video con- 
tent was requested, the Audio/Video Data Manager 6 
looks for this content 21 in the AudioA/ideo File Server 
Complex 7. If not available 32, the Locator obtains it 33. 

'0 If needed, a viewer (such as the ICAVS viewer 13) is 
transmitted to the client 26. The Streaming Traffic Pacer 
9 then streams the audio or video to the client 27. 



'5 Claims 

1. In a computer system, a method of distributing re- 
quested content data in a communications session 
between a server computer system and a request- 

20 ing client computer system, said method compris- 
ing 

a. determining characteristics of said client 
computer system; 

25 

b. determining characteristics of a communica- 
tion path from said server computer system to 
said client computer system; and 

30 c. automatically transmitting said content data 

from said server to said client computer system 
at a bit rate and in a format compatible with said 
client computer system and said communica- 
tions path. 

35 

2. A method as recited in claim 1 , wherein said char- 
acteristics of said communications path are contin- 
ually determined and said bit rate is dynamically 
modified to adapt to changing said characteristics 

40 of said communications path. 

3. A method as recited in claim 1 , further comprising 

a. maintaining status information as to content 
45 data and its format which is most commonly re- 

quested by clients; and 



b. storing latter said content data in said format 
in a cache for further distribution. 

so 

4. A method as recited in claim 1 wherein said format 
for transmission of said requested content data is 
selected from said characteristics of said client sys- 
tem; and said bit rate to be used is selected from 
55 said characteristics of said communications path; 
the method further comprising: 

transmitting software to said client computer 
system which downloads said software to enable it 
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to decode said content data in said selected format. 



quest for latter said content data. 



5. A proxy internet server for delivering content data 
to a requesting client computer system, said server 



a. means tor selecting a format for transmission 
to said client computer system in accordance 
with characteristics of said client computer sys- 
tem and a communications path to said client '0 
computer system; 

b. means for selecting a bit rate in accordance 
with characteristics of said client and said com- 
munications path; and *5 

c. means for transmitting said content data in 
said format and at said bit rate to said client 
computer system. 

20 

6. A proxy internet server as recited in claim 5, further 
comprising means for transmitting software to said 
client computer system which downloads said soft- 
ware to enable it to decode said content data trans- 
mitted in said selected format. 25 

7. A proxy internet server as recited 5 t wherein said 
means for selecting said format comprises a switch 
and a array of hardware encoders and decoders 
connected to said switch. 30 

8. A method as recited in claim 1 , wherein said char- 
acteristics of said client computer system are deter- 
mined by downloading a software module by said 
client computer system which automatically returns 35 
it characteristics to said server. 

9. A method as recited in claim 1 , wherein said video 
content is stored by said server in a plurality of for- 
mats if said video content was previously requested 40 
by said client or another client. 

10. A method as recited in claim 9, wherein one of said 
formats is a common format that is likely to be used 

by a plurality of requesting clients. *s 

11. A method as recited in claim 1, further comprising 
storing said requested content data at said server 
for possible future transmission to a requesting cli- 
ent computer system. so 

12. A method as recited in claim 1 , wherein said server 
searches the internet for said requested content da- 
ta if latter said requested content is not stored by 
said server. 55 

1 3. A method as recited in claim 1 , wherein content data 
is stored in a cache each time there is a client re- 
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